1. Field of the Invention
The present invention relates to an information processing method and apparatus for displaying an image in which a pointer image operated by an operator is synthesized.
2. Description of the Related Art
In virtual reality (VR), it is possible to see a three-dimensional virtual space from a viewpoint of an operator using a display, such as an HMD (head mounted display) or the like, and perform an operation in such an environment. In mixed reality (MR), an operator can see a real space and three-dimensional virtual space in a superposed state using a see-through HMD.
In a telepresence system, an operator can perform an operation as if he is actually present at a remote location, by operating a camera provided at the remote location and performing the operation while seeing an image taken by the camera.
In such VR, MR or a telepresence system, an operator sometimes performs an operation for a real or virtual three-dimensional space. For example, the operator moves a virtual object within a three-dimensional virtual space, or designates an object in a real thee-dimensional space and displays information relating to the object on a display. Conventionally, such an operation is performed using a three-dimensional input device, such as a magnetic sensor, an optical sensor or the like, that can perform three-dimensional designation.
In contract to methods for inputting an operation for a three-dimensional space using a three-dimensional input device, methods for performing an operation in a three-dimensional space using a two-dimensional input device, such as a mouse or the like, have been known. As an example of such methods, a method for performing input to a three-dimensional space by obtaining a z-coordinate value in the direction of the depth from a designated position on screen coordinates (x, y) obtained by perspective transformation of a three-dimensional space from a previously prepared table, using an input device, such as a mouse or the like.
In any of such methods using a two-dimensional input device, an input operation is performed by operating a pointer by a two-dimensional input device in an image obtained by perspective transformation of a thee-dimensional space. An example of VR application software for inputting an operation from a two-dimensional input device for a three-dimensional space in such a conventional input method will now be described.
The operator mounts an HMD 201 shown in FIG. 1. The HMD 201 provides the operator with a three-dimensional virtual space. The HMD 201 includes a head-position/posture sensor 202 that outputs the position and the posture of the HMD 201. The operator has a two-dimensional input device 101 shown in FIG. 2 in his hand. The operator can input two-dimensional values using a two-dimensional pointing stick 102. That is, as shown in FIG. 2, the operator can input values (x, y)=(0, 0)–(100, 100) depending on the direction and the angle of inclination of the pointing stick 102. The two-dimensional input device 101 also includes a first button 103 and a second button 104.
FIG. 3 illustrates an outline of a three-dimensional virtual space presented to the operator using such application software. A three-dimensional virtual space 300 comprises a rectangular parallelepiped 301. Within the rectangular parallelepiped 301, virtual spheres A 302, B 303 and C 304 are floating. The operator is also present within the rectangular parallelepiped 301. Reference numeral 306 represents a world coordinate system whose origin is at a corner of the rectangular parallelepiped 301. FIG. 4 illustrates how the virtual space 300 is seen from the operator's viewpoint.
FIG. 5 represents a picture frame 500 displayed in the operator's HMD 201. The picture frame 500 has a width comprising 640 pixels and a height comprising 480 pixels. A coordinate position on the picture frame 500 is represented, for example, as (x, y)=(640, 480) making the upper left corner of the picture frame 500 the origin. A pointer 501 is displayed on the picture frame 500, and moves by the user's operation of the two-dimensional pointing stick 102. The pointer 501 moves on the picture frame 500 in accordance with the amount of operation of the pointing stick 102. In FIG. 5, the pointer 501 assumes a position of (x, y)=(490, 80).
When the operator depresses the first button 103, an object present under the pointer 501 is designated. For example, in FIG. 5, the pointer 501 is displayed in a state of being superposed with the virtual sphere A 302. Hence, when the operator depresses the first button 103, the virtual sphere A 302 is designated. At that time, determination whether or not the pointer 501 is present on the virtual sphere A 302 can be performed, for example, by acquiring the z buffer value of the coordinates (490, 80).
The above-described three-dimensional input device using a magnetic sensor or an optical sensor is generally expensive. Furthermore, since the number of objects to be measured at a time is sometimes limited depending on the sensor being used, it is sometimes impossible to measure the input device using a sensor. In addition, for example, in a three-dimensional input device using an optical sensor, measurement cannot be performed if a body or an object is present between a measuring apparatus and an object to be measured.
The above-described problem is solved by using a two-dimensional input device, such as a mouse or the like. However, in the conventional method of moving a pointer using a two-dimensional input device in an image obtained by performing perspective transformation of a three-dimensional space, if the direction of the operator's line of sight is unstable, the designated direction of the pointer greatly changes in the three-dimensional space. In other words, since the designated direction is defined by a field-of-view coordinate system, the designated direction greatly changes in a world coordinate system when the field-of-view coordinate system is not fixed.
For example, when the operator mounts an HMD, the designated direction greatly changes even if the position coordinates of the pointer do not change on the picture frame, if the operator's head fluctuates due to the weight of the HMD. Accordingly, in order to exactly designate an object in a three-dimensional space, the operator must fix his head, resulting in pain for the operator. This problem will now be described with reference to drawings.
FIG. 6 is a diagram illustrating the relationship between a world coordinate system 306 and a field-of-view coordinate system 601. The position and the posture of the field-of-view coordinate system 601 in the world coordinate system 306 are obtained from the head position/posture sensor 202. An image plane 602 is arranged relative to the field-of-view coordinate system 601. A designated direction 603 of the operator is represented by an arrow connecting the origin of the field-of-view coordinate system 601 and the pointer 501.
FIG. 7 is a diagram illustrating how the operator's designated direction 603 changes when the direction of the operator's line of sight changes due to fluctuation of the operator's head. As shown in FIG. 7, when the operator's line of sight changes, the relationship between the world coordinate system 306 and the line-of-sight coordinate system 601 changes, to change the positions of the image plane 602 and the pointer 501, thereby greatly changing the designated direction 603 in the world coordinate system 306.
FIG. 8 illustrates a picture frame 801 displayed in the operator's HMD 201. A frame 802 indicated by broken lines indicates the position of the picture frame before the line of sight changes, i.e., at the time of the state shown in FIG. 6. As can be understood from FIG. 8, before the line of sight changes, the pointer 501 is displayed in a state of being superposed with the virtual sphere A 302. The position of the pointer 501 changes due to a change in the direction of the line of sight, and is displayed in a state of being separated from the virtual sphere A 302. Accordingly, in order to designate the virtual sphere A 302, it is necessary to again move the pointer 501. The same operation must be performed if the direction of the line of sight changes due to fluctuation of the head after the movement. That is, in order to perform a stable operation, it is necessary to fix the operator's head mounting the HMD 201, resulting in a pain for the operator.
Although the above-described example represents a case of VR application software using an HMD, similar problems will arise in MR application software or telepresence application software. For example, in a telepresence system, when a camera is placed at a remote location, the photographing direction of the camera sometimes fluctuates due to mechanical factors or a change in the external environment caused by wind or the like. The above-described conventional approach cannot deal with such a problem because it presumes a state in which the position of the viewpoint and the direction of the line of sight are fixed.